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AUTOMATIC ON-TBE-FLY FOCUSING FOR CONTINUOUS IMAGE 
ACQUISITION IN fflGH-RESOLUTION MICROSCOPY 

BACKnRnTTKTp np THE TNVFN77 r>|.T 
5 The present invention relates to microscopic imaging, and more 

particularly, to simultaneous image scanning at a variety of focal depths for 
improved automatic focusing during continuous imaging of a microscopic 
specimen. The invention is advantageous for rapid scanning of an entire 
microscope slide at high image resolution. 
10 The perfonnance of a scamiing cytometry system based on optical 

microscopy depends both on the scamring instrumentation's characteristics and on 
the specimen's characteristics. The instrumentation perfonnance criteria include: 
lateral resolution, depth of field, focus range, focus accuracy, focus ftequency 
response, slide scamiing rate, photometric sensitivity and dynamic range. 
Important specimen characteristics include tissue thickness and sUde flatness in 
relationship to the instrument's objective numeric apermre (NA). Biological 
specimens do not lie in a single focal plane across a microscope slide which 
complicates automated scamiing of the specimen for image cytometry using 
typicaUy narrow depth-of-field microscope optics. 

Adapting existing scanning systems to optical microscopy often results in 
a compromise between scamiing speed and image resolution. For example, web 
inspection systems with time-delay-and-integrate (TDI) sensors are usli in 
automated high speed visual inspection systems to detect and identify defects on 
continuously moving objects. The object in the field of view of the sensor is 
assumed to be flat and a 2D image is formed by synchronizing the Une frequency 
of the camera to the web movement. When the web or conveyor is not flat, image 
quahty can be degraded. Therefore, the depth of field is usually made large and 
carefully adjusted to include the largest variations in axial position. In these 
applications resolution is not a limiting factor and can be sacrificed to obtain laige 
depths of field and avoid the need for an autofocus system. When web inspection 
techniques are appUed to optical microscopy, as in the macroscopic 
implementations, large depths of field were ensured by the use of low NA 
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Objectives in order to avoid the need for autofocus. This is appropriate in low 
resolution imaging over limited areas, but requires depth of fields of on the order 
of 10 microns (um) or more over the length of a standard microscope slide. For 
higher resolution imaging, the depth of field is limited to about 1.0 Mm and 
5 refocusing at each field of view is required to improve image quaUty. 

Another exemplary scanning system is optical profilometry. In optical 
profilometry, a point source of light is imaged on the specimen. Best focus is 
deteraiined essentiaUy by minimum point size and a focus eiror detection system 
is used to measure surface topography. A number of different methods are used to 
1 0 produce an enor signal proportional to height variations from minimum point size 
and this error signal is used to servo control best focus. The height variations are 
then recorded as lens positions during scanning. The data show horizontal 
resolutions of 1 to 0.01 ^m and vertical resolutions of 0.5 to 0.001 pm. However, 
the surfece slopes of the specimen have a significant influence on the 
measurement accuracy. One Profilometer follows the specimen surfece very well 
at repositioning frequency of 100 hertz (Hz) with specimen height variations up to 
0.2 m- In open loop mode, the objective lens is fixed and the surface height is 
derived directly from the focus error. Open loop mode increases the measurement 
fi^quency, but a servo is required when the axial range is large such as with 
biological specimens. Since data is gathered one point at a time, this method is 
slow and apparently has not been ^plied to semitransparent specimens (like 
tissues) that can reflect from multiple surfaces. 

Accordingly, there exists a definite need for an automatic microscopic 
scanning system that can r^idly scan, at high resolution, an entire microscope 
sUde containing a biological specimen. The present invention satisfies these needs 
and provides further related advantages. 
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STJVfMAT?V nv J HE INVF.NTTON 
The present invention is embodied in an imaging apparatus, and related 
30 method, for fiilly automatic, high-speed, high-resolution image cytometry. The 
imaging apparatus iqjdates the focus many times across a conventional microscope 
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field Of View while moving the stage in a continuous motion at a constant velocity. 
The improved focus accuracy and allows performance of high resolution scanning 
image cytometry at high speeds and analysis of an entire microscope sUde in 
several minutes. 

Other features and advantages of the present invention should become 
apparent from the following description of the prefeired embodiments, taken in 
conjunction with the accompanying drawings, which illustrate, by way of 
example, the principles of the invention. 

SKB5F DESCRIPTION OF tvtp dr AWTMr.g 
no. 1 is a schematic diagram of a volume hnaging apparatus for 

continuous imaging of a specunen using imaging faces having adjustable focal 

depths, according to the invention. 

FIG. 2 shows, a sUdeimaged at three locations, of differing cell 

distributions, with adjacent images and graphs of focus index versus vertical 

position. 

FIG. 3 shows images of a cell's original flourencence image, processed 
image, and contour image at optimum focus, at positive defocusing and at 
negative defocusing. 

FIG. 4 is a three-dimensional plot of focus function curves for a single 
one-direction scan. 

FIGS. 5A - 5D are schematic diagrams representing a variety of 
predetermined spacings for the imaging faces, according to the present invention. 

FIG. 6 is a perspective view of an embodiment of a volume imaging 
apparatus for continuous imaging of a specimen using imaging faces having 
adjustable focal depths, according to the invention. 

FIG. 7 is an elevation view of the volume imaging ^paratus of HG. 6. 

FIG. 8 is a perspective view of a half portion of the volume imaging 
apparatus of FIG. 6, showing the mechanical strurture for supporting coherent 
fiber bundles that couple images from the imaging faces to a CCD sensor and 
circuit board. 
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FIG. 9 is a perspective view of a half portion of the volume imaging 
^paratus of FIG. 6, showing the mechanical structure for adjusting the relative 
focal dq)ths of the imaging faces, according to the invention. 

FIG. 10 is an exploded perspective view of a half portion of the volume 
5 imaging apparatus of FIG. 6, showing the mechanical structure for supporting one 
of the coherent fiber bundles and for adjusting the relative focal depth of the 
corresponding image face, according to the invention. 

FIG. 1 1 is a block diagram of a host computer, a timing sequence, remote 
sensors, and parallel autofocus circuits for focusing and generating and processing 
10 the volume image obtained from the volume imaging apparatus of the inventioiL 

FIG. 12 is a timing diagram showing signal between the timing sequencer, 
the remote sensors, the autofocus circuits and the host computer of FIG. 1 1 . 

FIG. 13 is a block diagram for an automatic positioning module for 
controlling axial adjustment screws to position the image faces to predetermined 
1 5 relative positions. 

FIG. 14 is a schematic view of an embodiment of a volume imaging 
apparatus for continuous imaging of a microscopic specimen using beamspUtter 
for providing images having adjustable focal depths, according to the invention 

FIG. 15 is a schematic view of a beamsplitter for use in a volume imaging 
20 apparatus, according to the invention. 

FIG. 16 is a schematic view of an embodiment of a volimie imaging 
apparatus for continuous imaging a microscopic specimen using several sensors 
positioned at different axial and lateral locations each corresponding to unique 
focal planes. 

25 

mj MlW DESCRIPTION OF THF, PREFERRFD RMBODTMFNT5^ 
The present invention provides apparatus, and related method, for an on- 
the-fly autofocus system for high-resolution nnage cytometry which keeps the 
image sharply focused during continuous stage travel. A volume camera 10 of the 
30 invention acquires an image volume is by concurrent optical sectioning of the 
specimen 12 with a dedicated imaging array 14 as shown in FIG, 1 . Imaging array 
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has multiple predetermined imaging faces 15 having adjustable optical path 
lengths to the focal planes 16 allowing simple adaptation to objectives 18 with 
different depths-of-field and magnifications. Automatically tracking focus with 
high precision over an area the size of a microscope slide is particularly 
5 advantageous because even a shde having a thin, densely populated cell 
monolayer are imperfect in thickness. 

The autofocus technique performs sharpness measiu-ements on images 
acquired at a sequence of foci in parallel in order to extend autofocus to 
continuous scanning. The sequence of foci at which images are collected for 
10 sharpness measurements represent optical planes in a magnified 3D specimen 
space. Normally, a 3D image is collected by sequentially moving the stage axially 
a small amount between planes in the stack of 2D images. However, it is also trae 
that the magnified image of the specimen (i.e., the region near the image sensor) is 
a 3D representation of the specimen space. Image quality can degrade if the sensor 
15 is placed too far fcom the nominal focus in this 3D space. For a Nikon 2Qx 0.75 
NA Fluor objective (160 mm tube length), however, experimental MTF 
measurements have demonstrated no change in cutoff frequency over a range that 
altered the fi*ont focus of the objective (i.e., the focus in the specimen) by ± 50 ^m. 
Thus, multiple sensors 22 can be placed at different axial positions (zl-29)in the 
20 image space over quite a large range without reducing optical quality. These 
sensors can then concurrently acquire the optical sections that make up a volume 
image. The design of a dedicated imaging array (the volume camera) is presented 
along with a model for tracking focus during continuous scanning. 

The volimie camera 10 is created by placing image sensors at different foci 
25 as shown in FIG. 1. In this configuration, the sensors may also be displaced 
laterally. When this adjacent placement is used, image (or focus mdex) 
registration is achieved through buffering. 

Since axial magnification is approximately equal to the square of the 
lateral magnification, subresolution axial sampling of the specimen space 14 is 
30 achieved easily. This configuration allows acquisition of axially displaced optical 
planes that lie near the normal detector position without degradation in optical 
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perfonnance. Tl.e axially displaced detector, are line array sensors synchronized 
to the stage veIocity24. We used EG&G (Sunnyvale, CA) TO 1096 which are 
1024 pixel X 96 line IDI CCD anays with a 13 x 13 pixels. TDI cameras 
mtegrate in proportion to the number of lines to increase signal-to-noise and 
senszt^vrty. The system was also designed to allow axial adjustment of the TDI 
sensors to obtain optimal axial sampling for objectives with differ^t depths-of- 
field and magnifications. 

The computational demand of calculating and adjusting focus dynamically 
IS absorbed by an array of parallel autofocus circuits that measure the 3D image in 
10 real time. 

A volume imaging apparatus having autofocus capabilities is disclosed in 
U.S. Patent No. (PCT Application No: WO 96/01438 published 1/18/96) titled 
AUTOFOCUS SYSTEM FOR SCANNING MICROSCOPY. Tie image 
processing and autofocus systems disclosed in the WO/091438 application may be 
1 5 applied to the present invention. 

With reference to FIG. 2. the first slide location (a) shows the effect of a 
low signal when cells are absent Tie second slide location (b) shows the image 
and focus index obtained with a ceU monolayer. The third slide location (c) shows 
overlapping cells. The typical problems encountered in image cytometry 
correspond to differences in cell density across a slide. FIG. 2 (b) represents the 
optmral densely populated cell monolayer and FIGS. 2 (a) and 2 (c) represent the 
opposite extremes of an area with no cells and one with many overlapping cells in 
a multilayer. The focus function curve of a low signal field (FIG 2 (a)) is marked 
by a spurious maximum, a sharp sidelobe. and noisy background Tie cell 
multilayer (FIQ. 2 (c)) shows that the system might become confused by the 
multimodal focus response and track a sidelobe rather than the main best average 
focus peak. A system designed for high speed tracking of focus over a lar^e area 
must work robustly even when there are too few or too many cells. 

When performing automated ceU scanning at high lateral resolutions 
features such as ceU pattern and texture can be important for cell classification, but' 
the discriminatory power of these features is highly dependent on focus. For 
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accurate measurements from microscope images, high numerical ^erture (NA) is 
preferred in order to obtain good lateral resolution. However, the depth of field is 
inversely proportional to the square of the NA and sharp focus becomes critical 
even at moderate NAs (^.5). With narrower depths-of-field, biological specimens 
5 do not lie in a single focal plane across the slide and automated scanning for 
image cytometry must be accompUshed by autofocus. Because higher resolutions 
also dramatically increase the amount of image data, the autofocus must be 
combined with rapid scanning. 

The influence of focus in feature extraction is demonstrated by three 
10 different images of the same object at different foci (a - c) in ¥10. 3. The first 
image (a) was taken with the focus set to a position before the optimum focus 
position. The second image (b) was taken with the focus set to the optimum focus 
position. The third image (c) was taken with the focus set to a position after the 
optimum focus position. The differences in foci are less than the depth of field of 
15 the objective. Each of the original digital images had the same post image- 
acquisition processing but the corresponding edge maps are different. There are 
also differmces in intanal pattern and these differences could cause cell 
classification errors, decrease repeatability, and increase subsequent diagnostic 
error. Additional processing can be ^plied to recover defocused images, but these 
20 conqjlex iterative algorithms are also slow and often cannot deblur without some 
error. Real-time autofocus that maintains image sharpness precisely, therefore, is 
advantageous over algorithmic correction techniques. 

As shown in FIG. 4, the focus curve will follow the curvature of the 
specimen features as the slide is scanned. The horizontal sanqjle size is 650 nm 
25 and the focus measurement area is 3 12 pm (V) x 325 pm(H) with a 20x 0.75 NA 
phase contrast objective. 85 focus positions form a single focus function curve for 
each slide position. 

The focal planes 30 can be configured m a wide variety of predetermined 
relatives spacings. For example, as shown in FIG. 5A, the planes are configured 
30 in a staggered arrow configuration. Further, as shown in HG. 5B, the planes can 
be in a curved configuration with the planes at one end of the imaging planes 
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being more widely spaced in focal depth than the plane at the other end. 
Alternatively the imaging planes may be more widely spaced at the ends and more 
closely spaced in the middle, as shown in FIG. 5C. Likewise, the imaging planes 
may be configured in a V- or an invated V-shq)e, as shown in HG. 5D. As 
5 shown in FIG. 5E, another configmation for the focal planes is a W sh^e. 

One embodiment of a volmne imaging apparatus 10 or camera of the 
invention is described with reference to Figs. 6-10. hi this embodiment, unages 
&om adjacent imaging planes are transferred to separate detectors 32 by coherent 
fiberoptic bundles 34 having image faces 36 that are placed at predetermined 
10 image plane locations. The mechanical stracture of the volume camera allows for 
relative axial adjustment of each of the image feces. The volume camera has a 
bottom plate 38 that is mechanically coupled to a top plate 39 by four rigid 
support rods 40 that are fixed to the top and bottom plate roughly in the center of 
the area between the four rigid support rods. The top and bottom plates are shown 
1 5 cutout fiom a rigid support box that holds them fixed in space. 

The present embodiment of flie invention has nine fiberoptic bundles, 
corresponding support stractures and imaging seaisors. The nine bundles allow 
placement of the nnage faces over a 2 fmi range in 0.25 m increments. More or 
fewer fiber bundles may be used and smaUer or larger axial steps between the 
20 unage feces may be used in accordance with the invention. 

Each coherait fiberoptic bundle is supported by a base plate 42. The base 
plate has closely fitting holes 44 that allow the base plate to sUde along two of the 
rigid support rods. (FIG. 10) Each base plate also has a threaded hole 46 for 
receiving an adjustment screw 48-1 to 48-4. Each adjustment screw 48-1 to -4 has 
25 80 turns per inch for making micrometer adjustments to the base plate position. 
All of the base plates, except the base plate adjacent to the top plate, also have a 
bearing hole 50 for receiving the end of one of the adjustment screws 48-1. The 
top plate includes a bearing hole and additional adjustment out for receiving the 
adjusnnent screw for adjusting the location of the top base plate witii respect to the 
30 top plate. The remaining adjustment screws 48-2 to -4 are configured between 
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base plates to allow adjustment to a variety of predetennined face plate 
configurations. 

Each adjustment screw may be driven manuaUy or automatically by a 
motor controller as discussed below with respect to FIG. 13. Generally, the 
selected miagiiig 36 fece configuration wiU coirespond to a particular objective 
type and specimen 14 under lamination. 

The camera 10 includes CCD sensors 32 that are coupled to the imaging 
plane fiberopticaUy by bonding the imaging fiber bundles 34 (IFB) directly to the 
CCD devices. The fiber bundles translate the image plane to the surface of the 
CCD. Each CCD sensor is coupled to a circuit board 50 that is attached to the 
corresponding base plate by a mounting bracket 52. The objective 1 8 lens projects 
the image of the specimen onto the entrance face of each IFB, and the bundle 
transmits the intensity distribution of this image to the other end face and onto the 
sensor. The branching shape of the array was devised with IFBs of the same 
length so that image transmission properties are the same. The bend in the fiber is 
set to allow the cameras to fit and sUde axiaUy in a small camera package. 

Axial sampling can be controlled by shifting the input fiber faces axially, 
and the lateral separation of the fiber bundles 34 produces delays between the 
acquisition of each field of view. The series of fiber bundles faces 36 can be 
shifted to create a pattern at the imaging face (e.g. with a linear or V'-shaped 
staircase) to sample the image volume. 

The system for implementmg the autofocus adjustments and for pitjcessing 
the images received by the sensors 22 are described with respect to FIGS. 1 1 and 
12. 

A more detailed description of the image processing methods and the 
autofocus methods is described in U.S. Patent No. 5.790,710 and in U.S. Patent 
No. (PCT AppUcation No. WO 96/01438). 

The processing and control system 60 includes a host computer 62, a 
timing sequencer 64, parallel autofocus circuits 60 and the remote sensors 68. 

The timing sequencer controls the video acquisition and the focus 
measurement by triggering the rem te CCD sensors and setting the size of a focus 
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measurement virtual frame interval. A virtual frame is made of a number of lines 
to include in the focus measurement and the length of a window in each horizontal 
line. These values, number of lines and horizontal window duration, are 
programmable. A horizontal hne is made of an active portion and a blank region. 
5 The total length of this line is the integration time and is piDgrammable. During 
the integration period the CCD sensor elements collects light from the projected 
image. The integration time value is a tradeoff between the speed and the 
specimen illumination conditions. The sections in the timing sequencer include: 

a) Master Clock 70: Reference clock used to generate all logic timing 

10 signals 

b) Integration time 72: Times duration of a video hne 

c) Line start 74: Pulse at the beginning of a new video line 

d) Frame size 76: Pulse at the end of a programmable number of lines 

e) Horizontal window 78: Signal part of the active video for focus 
15 measurement Helps ehmioates transients produced in the filters of the focus 

measurement circuit 

f) Driver 80, receiver 82: Used to send or receive the high-speed logic 
signal over/frx)m large distances. 

The remote sensors 68 generate the video signal of the volume camera. 
20 The sections of the remote sensors include: 

a) Scan controller. 84: Uses the local clock, or an external one, to 
generate the clock phases required by the CCD sensor. Also, signals to the exterior 
when the active portion of the video is occuning. 

b) Local clock 86: Reference clock of the CCD. 

25 c) TDI CCD 88 : Sensor that converts the hght to charge 

d) Signal processor 90: programmable gain, and digitizer; process the 
output of the CCD create a direct high performance digital volume image that can 
be acquired and viewed independently. It also creates the analog video signal that 
is used by the focus measurement circuits. 

Receiver 92 and transmitter 94: Receive or send the high-speed 
logic signal from/over large distances. 
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The parallel autofocus circuits 66 perfoim the continuous focus 
measurement on several video inputs by using an array of autofocus circuits in 
parallel. Each autofocus circuit requires two logic signals fiom the timing 
sequencer 64: frame and window. The frame signals when to start a focus 
5 measurement interval and when to provide the recent measurement values to the 
host computer. The line window is a pulse that allows the focus measurement 
circuit to consider the video input for calculation. This pulse duration is valid 
during a portion of the active video. The sections of the autofocus circuits 
include: 

10 a) Programmable delay and timing control 96: Delay the 

measurement and generate the control signals that the focus measurement circuit 
requires: Integrate, reset, and enable. Also provides an interrupt signal to the host 
computer 62 when a new focus calculation is ready. 

b) Focus measurement circuit 98: Produces two output voltages that 

15 correspond to the focus index and the average illumination of the virtual video 
frame measured. 

The logic control signal groups start, frame_start line_window and 
vaIid_video, are grouped in terms of its fimction and individual signals can be 
active independent of the other signals in the same group. The timing generator 

20 activates these so that camera boards can be started at different times if desired, 
and the focus measurement is synchronized to its corresponding remote sensor. An 
initial goal is to measure focus at the same spatial region (field of view), even 
though, the sensor may be displaced laterally and a buffer of the measured values 
is necessary to compensate for this delay. 

25 FIG. 12 shows a timing diagram of the signals between the timing 

sequencer 64, the remote sensors 68, the autofocus circuits 66, and the host 
computer 62. After the Interrupt_l pulse appears, the host computer reads the 
focus index FI_1 and the average illunMnation AI_1 values from the autofocus 
circuit 1 . The same timing is vahd for the other channels but they may appear 

30 asynchronously. Digital auto-gain signals may also be provided as shown in U.S. 
Patent AppHcation Serial No. 08/796,196. 
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Much of the computational demand of calculating focus dynamically is 
absorbed by an amy of parallel autofocus circuits 66 that measure the focus 
indices of each plane of the 3D image in real time. The block diagram of 
autofocus system is shown in HG. 11. This system contains the CCD and 
autofocus electronics. The ou^ut of this system (the focus indices) are read by the 
computer 62 which in turn adjusts the focus with a piezoelectric objective 
positioner (Polytec PI, Costa Mesa, CA). A high performance, low cost analog 
circuit that measures the focus index and the average illumination in real time has 
been modified for TDI line scan video and replicated for each CCD sensor in the 
multiplanar array. A timing sequencer 64 works as the central controller of the 
parallel autofocus circuits 66 and the cameras 68 of the array, synchronizing focus 
measurement and image acquisitioa The TDI signal consists of video for each 
line punctuated by horiaxMital blanks between lines. A line counter provides focus 
index timing previously provided by the vertical blanks of a video signal. The 
15 sequencer 64, which sets the integration time of the CCDs, also contains a 
programmable line counter that creates a virtual fi^e to provide the focus index 
integration interval. The continuous focus measurement circuit 98 resets at the end 
of each virtual fi^me intwval and generates an intemq)t to the host computer 62 
for AfD conversion, calculation of best focus, and piezoelectric refocusing. 
20 With much of the autofocus conq)utational burden bom by the parallel 

autofocus circuits 66, the computer 62 can be programmed to execute ahnost any 
autofocus algorithm. The algorithm can preferentiaUy weight present values of 
focus over past values and utilize predictors for iqjcoming focus indices. This is 
different fi-om a conventional negative feedback loop. Because calculation of best 
25 focus fi-om sharpness measurements (or focus indices) does not require an iterative 
algorithm for comparison at successive trial positions, it can be carried out in real- 
time for rapid focus adjustment 

These rapid conq)utational methods make the 10 ms response of the 
piezoelectric objective positioner the slowest component in the focus 
30 measurement and update sequaice. The volume camera and continuous autofocus 
system will calculate and update the focus position at a rate of 100 - 1000 Hz 
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during scanning. With a relatively flat specimen, the scanning speed ultimately 
may be limited by the video rate of the TDI cameras. With a programmable 
number of video lines in the virtual frame (focus measurement window) the 
combination of focus positioning response and the focus index signal-to-noise 
5 ratio will determine the actual focus update fi^uency and the ultimate focus 
accuracy. 

Experience with mcremental scanning has shown that focus tracking can 
be lost when the focus test range is too narrow or a field is completely empty. As 
the virtual video frame is made smaller (a sub-region of the microscope FOV), 
10 local details (e.g., cells as well as surface irregularities) can increase axial 
variations and accentuate gradients so as to demand faster focus positioning 
response during continuous stage movement. Conversely, with larger virtual video 
frames, areas of high slope are averaged and spatial gradients diminish in 
proportion to the number of video lines in the virtual frame. The parameters of the 
15 test range can be determined for each specimen and objective lens conqjletely, but 
void fields (with no image content for focus measurement) could still pose 
substantial problems. Recent experiments with the autofocus circuit of the 
invention, however, has shown that debris much smaller than a single cell is 
enough for accurate focus tracking. In combination with linear prediction, this 

20 high focus measurement sensitivity allows accurate tracking over small areas that 
lack even small amounts of debris. 

In order to set the volume camera parameters to maximize accuracy and 
tracking stabihty, the focus function curve (FIG. (2a-c bottom)) and depfli of field 
may be used to guide in setting the spacing between optica] sections. Additional 

25 considerations for the spacing include the mininniTr spacing between optical 
sections that produces substantial differences in the sharpness measurements, and 
the range required to cover focus ftmction width. The focus search range may be 
ad^ted to fit an appropriate portion of the focus function curve range and 
variations frx>m specimen flatness. Focus fimction filter parameters (e.g., 

30 bandwidth, central frequency, and attenuation) can also be adjusted to achieve the 
best focus fimction response. 
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The events of a scan witb a volume camera consisting of N sensors are: 

a) At the end of each virtual ftame, an intemipt signals the computer 
that a new focus measurement has been completed for one of the planes. The 
outputs of the autofocus circuits can be transmitted to the computer 

5 asynchronously and buffered fliere prior to best focus calculation. 

b) All N focus measurements are acquired and buffered for one virtual 

fiame. 

c) On each buffered set of data that corresponds to one virtual ftame, 
best focus is calculated (e.g. using power-weighted average) and the lens 

1 0 positioned to best focus. 

The maximum time delay between the measurements corresponding to the 
same field of view is 

_ V,fAN-l) 

where M is the magnification, f, is the spatial sampling fiequency of the 
15 microscope-camera system, and F, is the line rate of the CCD sensor, and V, is 
the virtual video fiame length given by 

where is the number the lines in a focus measurement window and D 
is the pixel size. 

20 The lateral displacement of the sensors makes it possible for two or more 

sensors to sample the same, or nearly the same focal plane, if the slope of the 
specimen matches the slope sensor array. For this reason and other considerations, 
it may be that a simple linear staircase is not the best arrangement of the foci for 
the sensor array. 

25 The fiamework of a simultaneous multiplanar miage acquisition model 

that combines specimen and the volume camera characteristics has been 
developed to evaluate strategies for tracking focus. HG. 4 shows the femily of 
focus fimction curves for a single scan in one direction over a slide (witii 
monolayers of NIH 3T3 cells). The ridge of this 3D femily of curves corresponds 
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to the series of best foci in the scaa The shape of the focus response depends on 
the optical system settings and the specimen. The number of foci tested for 
autofocus in our previous experiments varied from 7 to 11, but the curves in 
FIG. 4 utilized 85 test foci to more accurately delineate the shapes of the focus 
5 fimction curves. These focus indices, collected from incremental scanning, create 
the virtual test specimen for testing various focus tracking algorithm. 

The use of focus functions with points from many optical planes provides 
an efi&cient way to simulate any biological specimen by reducing the amount of 
data required to store long lateral sections through the specimen sample. This 
10 specimen model will be useful primarily for focus assessment of the sample under 
observation. By comparison of the continuous scanning best foci with the 
incremental best foci, the performance of various algorithms and sensor positions 
can be measured. 

Table 1 presents a matrix of focus measures that correspond to the 
15 distribution of optical planes in the volume camera. These focus values and their 
coordinates (z, x) are used to reconstruct the focus function behavior of the 
specimen and to calculate best focus position for subsequent focus adjustment. 

Table 1 . Distribution of the volume camera focus measurement planes. 
F(Z, X,) F(Z, XJ ... F(Z, XJ 

F(Z,-,X,) F(Z„X,) ... F(Z„,XJ 

F(Z„X,) F(Z3,X,) ... F(Z„XJ 

F(Z„ X,) F(Z„ X,) ... F(Z„ XJ 

10 Where j, k, and m, which represent the number of axial positions in a 

specific field of view, do not necessarily correspond to equal spacing. X„^, > 
X^ , but Z„ is not conditioned with respect to Z„_, . The F(Z,X) values 
represent the focus measure of imaging planes at axial position Z and lateral 
position X in the field of view. X represents the lateral spacing between sensors. 
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The number of planes, the spacing between them, and the search range 
required to track focus depend on the thickness of the specimen magnified onto 
the image volume, the slope of the specimen, and the speed of scanning. 
Therefore, the application and the specimen will determine the number, spacing, 

5 and extent of the optical planes for best tracking performance. 

Focus tracking can also be limited by the axial gradient of the specimen 
and its velocity. In FIG. 4, there is an mitial z gradient in the best focus ridge that 
then flattens out at large x. If this gradient is too large, the system wiU not track 
focus at high speed. A second part of the mafliematical model considers the lens 

0 positioning response during tracking. If the velocity in the direction of stage 
motion is Vs = dx/dt then it can be expressed in terms of system parameters by 




The length of the specimrai covered in a focus measurement interval is 

Js 

15 where is the number of Unes in the virtual ftame. The specimen 

gradient, G = dz/dx, must not be so large as to exceed the objective lens positioner 
response at a given stage velocity. Therefore, in a distance AX, the maximum 
specimen axial shift AZ that can be tracked accurately is approximated by 
combining equations 3 and 4 as 

20 \AZ\<^^^ 
' Ft 

where Zr is the focus search range and is the settling time of the 
positioner. This last expression is a good qjproximation independent of possible 
errors in the best focus algorithm and positioning; that is, with assumption that 
calculated best focus equals true focus and that the piezoelectric objective 
25 positioner is perfectly accurate. Table 2 shows the specimen detail that can be 
tracked accurately for some typical values. For these calciilations more 
conservative of 20 ms was chosen (instead of the specified 10 ms for the 
piezoelectric positioner) in order to allow for the other possible errors. 
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Table 2. Specimen height and slope to tracked at diflferent magnijScaticns. 



M 






AZ/AX 




lOX 


10.6 


41.6 


0.25 


1.20 


20X 


10.6 


20.8 


0.51 


0.650 


40X 


10.6 


10.4 


1.02 


0.325 



Assuming typical values of Z, = 100 fan, t, =20 ms, LR = 15 KHz, and FS = 32 lines. 

Autofocus accuracy depends on virtual field size and the speed of objective 
positioning: a large virtual fiame reduces flie accuracy in tracking the specimen, 
5 and slow objective positioner response averages detail in the axial dimension. 

The axial position of the image feces 36 of the fiber bundles 34 may be 
adjusted by an automatic positioning module 102 shown in FIG. 13. The host 
computer 62 is coupled to a motor controller 104 associated with each image face 
and sends signals to the motor controller indicating the desired position for the 
10 corresponding image fece. The motor controller actuates a motor 106 that, is 
mechanically coiq)led to the corresponding adjustment screw 48. As shown in 
FIGS 6 - 10, the adjustment screws control the axial position of the image faces of 
the fiber bundles. 

FIGS. 14 and 15 shows a volume image ^paratus 10', according to the 
15 invention, that uses beamsplitters 112 to obtain the multiple planes 114 in the 
image volume. In this embodiment, the optical path lengths (zl to z4) to each 
sensor 116 are different (by AZ) and can be adjusted by moving the sensors 
themselves. The imaging ^aratus has three beamsplitters for generating four 
image paths. The beamsphtters are 50% splitters witii 50% of the light being 
20 reflected fiom the surfece of the beamspUtter and with 50% of the light being 
transmitted through the beamsplitter. The first beamsplitter separates collimated 
light fi-om the microscope optics 1 18 into two paths. Each of the two paths are 
split by a respective beamsplitto- generating a total of four paths. Each of the four 
paths could be further split into eight or more paths in accordance with the 
25 invention. 
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Each beamsplitter 112 splits the path once, and the total number of 
beamsplitters required depends on the number of planes of acquisition. In some 
paths, an additional mirror (not shown) will be required to maintain identical 
• miiToring at the sensors. 
5 Another embodiment of a volume camera 1 0" of the invention is shown in 

FIG. 16. The specimen 122 is magnified by the objective lens assembly 124 and 
multiple planes within the specimen slide are in focus simultaneously. The lens 
assembly magnifi es the specimen into the image space 126, where it is captured 
by several sensors positioned at difference axial and lateral locations. Each sensor 
10 in the array collects a unique focal plane and each coiresponding image contains 
different in-focus and out-of-focus information. 

While the foregoing has been with reference to specific embodiments of 
the invention, it will be appreciated by those skilled in the art that these are 
illustrations only and that changes in these embodiments can be made without 
1 5 departing Scorn the principles of the invention, the scope of which is defined by the 
appended claims. 
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What is claimed is: 

1 . Apparatus for generating a volume image, comprising: 

a plurality of imaging devices positioned at adjacent image planes wherein 
at least one image device is configured to be adjustably displaced at a 
5 predetennined depth of focus. 

2. Apparatus for generating a volume image as defined in claim 1, further 
comprising an image forming device coupled to the plurality of imaging devices 
for foiming a plurahty of electronic representations representing images at 

1 0 respective image planes of the plurahty of image planes. 

3. Apparatus for generating a volume image as defined in claim 1, further 
comprising a focusing device that selects one of the imaging devices associated 
with an optimum depth of focus for generating an image representation of an 

1 5 image obj ect of the volume image. 

4. A method for volimie imaging using a plurality of imaging devices 
positioned at adjacent image planes wherein at least one image device is 
configured to be adjustably displaced at a predetennined depth of focus, the 

20 . method comprising the steps of: 

orienting the plurality of imaging devices such that the imaging devices 
observe an image object in an image object plane; 

using the plurality of imaging devices, obtaining a plurality of images of 
the image object, each image of the plurality of images obtained in a respective 
25 imaging device; 

converting the plurahty of images into a plurahty of electronic image 
representations; and 

storing the pliuality of electronic image representations. 
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5. A method for volume imaging as defined in claim 4, wherein the image 
object is continuously imaged by translating the image object with respect to the 
imaging devices in a direction parallel to the image object plane. 

6. A method for volume imaging as defined in claim 5, wherein the 
electronic representations are divided into image fi^es in response to a counter 
signal. 
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FIG. 3 
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FIG. 9 
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